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Themes selected for this YLEM Journal issue encompass issues related to digital art and information visualization that support both research 
and artistic activities. For example, a visual representation of the search results can support understanding pi the web search, especially 
when there is a need for viewing multiple search results. As it was stated by Robert Treolar and Mikael Jern 1) problems arising in many 
research and industry application fields relate to knowledge construction, management and visualization. A real-time data analysis can be 
made by combining statistical analysis with visualization and techniques for interactivity. Visualizations can reveal information embedded 
in semantic web graphs such as topography or clusters. Data evaluation is supported in such research areas as neural networking, semantic 
web visualization, knowledge discovery and visual data mining by coupling the visual and statistical analysis into one tool. 


On the other hand, an access to the computer art software packages eliminates tedious hand skill exercises. In a study "Creative Performance: 
Does the Computer Retard Artistic Development?” Tammy Knipp 2) examines the role of perceptual-motor skills put into effect by tedious 
tactile/kinesthetic hand skills in improving cognitive development and neural organization. Such exercises, which may be eliminated by the 
use of computers, may be crucial for the development of intellectual skills and conceptual thinking. Tammy Knipp applied a Rey test for 
the brain damaged subjects to examine the level of perceptual performance of subjects in the form of their ability to recreate a complex 
figure from memory. Students (beginning, intermediate, and advanced artists) stated their level of knowledge or skills with computer art 
software packages, then they drew the figure from memory. Performance scores did not rise with the declared knowledge of software. The 
result interpretation discusses the role of perceptual development: a possibility that tedious tactile/kinesthetic hand skills which may be 
eliminated by the use of computers may be necessary for improving cognitive development and neural organization. The author propounds 
that visual/haptic exercises executed by the designer's hand foster perceptual-motor skills which are crucial for the development of 
intellectual skills and conceptual thinking. 


In this issue we present several connections between art and visualization technologies. 


-- Zeng, Mehdi and Gough discuss in a paper entitled "Shape of the Story: Story Visualization Techniques” the possibilities of incorporating 
knowledge of human-to-human interaction to story visualization technology by using natural language processing and 3D computer 
graphic techniques. Picture books, theatre, television, and movies extract abstract information of the story into concrete visual scenes to 
enhance experience and evoke emotions.The authors approach aims to bridge the gap between scripting/storyboarding by a non-technical 
creative writer and rendering characters and scenes that are normally created by a graphics specialist. 


-- Concepts related to photorealistic 3D renderings and interactive visualizations are often derived from an analysis of traditional media. In 
a paper “Learning from an analysis of traditional media: cognitive perception addressed by traditional painters,” John Counsell tells about 
hybrid representations in VRML and video. He discusses how perceptual artists used isometric projections and techniques closely related 
to cognitive perception (normal visual perception, not fully based on Euclidian perspective) rather than either perspective or orthographic 
projections. 


-- A study of a creative process in a collaborative project between an HCI team and a Japanese artist Yasunao Tone has been described in 
"Macaroni Synthesis: A Creative Multimedia Collaboration” by Ernest Edmonds and Linda Candy. A study of the three viewpoints -- the 
technologist's, the observer's, and the artist's perspectives supports a search for the creative use of technology. 


-- Thomas Panas, Rebecca Berrigan, and John Grundy proposed a 3D visual approach to depict software production cost related program 
information to software maintenance costs. 


Materials for this issue come from the Proceedings of the Seventh International Conference on Information Visualization (IV), London, 16- 
18 July 2003, IEEE (The Institute of Electrical and Electronics Engineers, Inc.) Computer Society, Los Alamitos, CA. Works of scientists and 
artists result from a cross-section of disciplines related to visualization and computer-based information: medicine, engineering, media, 
commerce and art. 


At the International Conferences on Information Visualization and Graphics, computer-based visualization techniques and applications 
are discussed yearly, with a wide multidisciplinary scope including art, visual representation of knowledge and developing 3D models, 3D 
virtual environment and augmented reality, web search result visualization for viewing multiple search results, collaborative information 
visualization environment, visual data mining with artificial neural network, and semantic web visualization. Noticeable stress put on 
the cooperation between artists, scientists, and people from industrial and commercial companies makes this event especially significant 
and valuable. The selection of papers and presentations shows that the image quality gains in importance as a vital factor in information 
visualization and visual communication. 
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The International Symposium on Digital Art and Online Gallery (IV D-Art) is accompanying each IEEE IV Conference. Works sent by artists 
from around the world are exhibited online. Also, the Symposium and Gallery of Digital Art (CGIV D-Art) will accompany the International 
Conference on Computer Graphics, Imaging and Visualization (CGIVO5), July 2005, Beijing China. 


International Symposia on Digital Art and Online Galleries (IV D-Art) and Symposia and Galleries of Digital Art and Online Galleries (CGIV 
D-Art) are yearly events, and a Call for Participation for the 2005 events are posted at the Internet: 


http://www.graphicslink.demon.co.uk/IVO5/DART.htm 
http://www.graphicslink.demon.co.uk/IVO5 


http://www.graphicslink.demon.co.uk/cgiv05/DART.htm 
http://www.graphicslink.demon.co.uk/cgivO5 


All enquiries should be addressed to Anna Ursyn ursyn@.unco.edu, or Anita D’Pour IVO5_CGIVO5@graphicslink.demon.co.uk. 


(1) Robert Treolar and Mikael Jern, Linkoping University Sweden, Applied Visual User Interface Technique in Knowledge Management, Proceedings of the 
Seventh International Conference on Information Visualization IV 2003, IEEE Computer Society, Los Alamitos, CA, p. 2) 

(2) Tammy Knipp, Florida Atlantic U., Creative Performance: Does the Computer Retard Artistic Development? Proceedings of the Seventh International 
Conference on Information Visualization IV 2003, IEEE Computer Society, Los Alamitos, CA, pp. 621-625. 


ANNOUNCEMENTS: 


YLEM Forum: Art Enlivening Biology 
Tuesday, March 15, 2005 7:30 pm McBean Theater, Exploratorium, 3501 Lyon St.,, San Francisco, CA 94123 
This is a FREE event, open to the public and wheelchair accessible 


The YLEM Forum, "Art Enlivening Biology,” features Jo Hanson, whose work in combining art and ecology is legendary, Julie Newdoll, a 


painter combining diagrams of biologically active molecules, images from mythology, and Jonathon Keats’ “misapplications of science.” 
Due to the retirement of our great benefactor, Larry Shaw at the Exploratorium, we'll no longer be able to hold our forums at the 
McBean Theatre. We have shifted our program to the RX Gallery, who are generously donating their excellent space to us. It is very 
conveniently located just a few blocks west of Union Square in downtown San Francisco, at 133 Eddy Street. 


Life Illuminated; From Atoms to Organisms” bio-art show 


YLEM announces plans for an exhibit which features the artwork and perspectives of both artists and scientists, imagining what we can- 
not see and exploring what we can about our biological world. For more details, go to http://www.ylem.org. 


Call for Entries - Juried YLEM Members Show at Stanford University 


YLEM announces plans for an exhibition of two dimensional artworks by YLEM Members. All YLEM members are encouraged to apply. 
This show will be juried by exhibition staff at Stanford University, so an application of entry is not a guarantee of exhibition. For more 
details, go to http://www.ylem.org. 


Call for Entries - YLEM Members Exhibition 


YLEM is presently contacting exhibition spaces for a Members Only Exhibition. This exhibit will be juried by YLEM staff and is not limited 
to two dimensional work as with the Stanford Exhibition noted above. For more details go to http://www.ylem.org 
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Macaroni Synthesis: A Creative Multimedia Collaboration 


by Ernest Edmonds and Linda Candy 
Introduction 


The Creativity and Cognition Research Studios (C&CRS) were 
established for the purposes of developing new art and technology 
projects and to conduct research into the creative process. In the 
COSTART Project 1998- 
2003 an_ innovative 
approach to the 
study of technology- 
based art founded on 
practice-led research 
methods has _ been 
developed. The work 
centers on the practice 
of art making. In this 
paper, we report on the 
particular innovations 
associated with one 
of the residencies; 
that of Yasunao Tone, 
an artist who was 
awarded the 2002 Ars 
Electronica Golden 
Nica prize for Digital 
Music. Tone has been 
changing calligraphic 
characters into images, 
then the images into 
sounds. The artists and 
technologists explored 
sound representation in 


real-time interaction 
between image and 
sound. 

The COSTART Project 
developed a new 
approach to the 


study of technology- 
based art founded on 
practice-led research 
methods. The artist- 
in-residency is at the 
heart of the process 
but it is a residency 
that differs in a number 
of important respects 
from the conventional type. First, it is a fundamental part of the 
approach that a residency support team with specialist knowledge 
about both technology and art practice is established. The team 
has expertise in many forms of computer systems, programming 
languages and devices but equally important is their personal 
involvement in music and visual arts practice both as researchers 
and practitioners. In addition, the environment into which the artist- 
in-residence comes is designated primarily for their use rather than 
on a guest access basis. Artists are invited to submit a proposal for 
a new project and to specify both artistic and technical needs. The 


Figure 1: Yasunao Tone Performance, 2003 
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research team carries out feasibility studies into each proposal as to 
whether the requirements can be met and whether it has sufficient 
challenge technically. The research exercises involve collecting data 
about the collaborative creative process that takes place in the core 
event, a five-day residency. This period is followed up with further 
development work in preparation for the exhibition of works that 
arise. The study data is analysed ready for publication in reports 
and papers. It is central 
to the approach that 
the research process 
and the creative practice 
take place in tandem 
and that the experience 
for the artist represents 
a realistic — situation. 
See Explorations in Art 
and Technology by the 
authors. 


A Residency Study 


Yasunao Tone is often 
associated with the 
Fluxus group that began 
in the 1960s. Throughout 
the 1960s and 1970s 
Fluxus evolved into 
an international art 
movement, notorious 
for its refusal to accept 
traditional ideas 
about art, culture and 
authorship. Tone has 
pursued a totally new 
relationship between text 
and sound: Molecular 
Music is (1982-5) is the 
earliest experiment as 
such. The piece is based 
on the poems written 
in Chinese characters 
including three poems 
from the Tong Dynasty as 
well as the 8th Century 
Japanese poem. The 
piece employs the sound- 
generating system that 
includes light sensors 
attached on the film screen and oscillators connected to light 
sensors, so that the film projected on the screen creates varying 
sounds in accordance with the specific arrangement of the sensors 
and the changing brightness of the projected images. 


Yasunao Tone's process for working with the conversion of calligraphic 
drawings into sound is very time-consuming involving as it does 
changing first Chinese characters into images and then the images 
into sound. Such time scales restrict the use of real-time interaction 
with his work. The aim for the residency project was to explore the 


possibilities of sound representation in real-time. The intention was 
for the development of a program for performance that could focus 
on the interaction between image and sound. The instrument that 
was developed made it possible to do live performance in which, 
instead of having to transform the text into images with pictogram- 
like Chinese characters, the artist draws calligraphy on the board 
and transforms any text into sound. 


The concern throughout the residency was how working with an 
artist of this nature influenced the process of making the work, and 
how a rejection of prescribed methods, tools and structures changed 
the way in which software 
systems were developed. 
This activity was further 
complicated because of the 
apparent divide between 
the work and the means of 
making the work. So there 
was no point at which we 
were making simple tools 
or technical solutions that 
would later be used by the 
artist to produce the work 
itself. In the pre-residency 
discussions, it had seemed 
that it would be interesting 
to make data about his 
drawing gestures available 
computationally. The idea 
was that he would use the 
Soft-board in our studios. 
This device looks like an 
ordinary whiteboard but 
has the additional capability 
to detect the position of 
a pen on its surface. This 
information is output via a 
serial cable and can be used 
to keep a dynamic record of 
developments in a meeting, 
for example. 


When it was suggested to 
Yasunao that a variation 
on granular synthesis to 
generate sounds could be 
used, he said that his idea 
was more like ‘macaroni 
synthesis, because it already 
had a form’ Tone explained 
what he meant by that: 


Figure 2: Yasunao Tone Performance, 2003 


Y: Oh! | have a piece called “Molecular Music" which... the title 
itself is "My Dissatisfaction with Computer music”. Computer Music 
is a kind of... started from atom. Yes... so | started from molecules 
not from atoms. And this time | told Mark “This is a macaroni"... its 
developed through the granular synthesis. Granular synthesis is... he 
obtains certain tiny bits of a wave form and in the building, so doing 
you have to make a pitch how you make yours, you have to do the 
same thing as | know, take academic computer music. So | don't like 
that. So instead of grain, | use macaroni. 


C: Well there's some cooking required after that. 

Y: Yes. Cooking. 

C: Is it cooked macaroni first, or the basic ingredients? 
Y: Yes. Basic Ingredients. 

C: The performance is the cooking? 

Y: Right. 


When the team try to imagine a system that transforms the 
movement of a pen into a sound they almost inevitably started to 
think in a certain way - to define a complex system of conditions, 
choices, relationships, behaviors, where a given input is mapped to a 


particular parameter and a certain kind of intervention has a certain 
kind of result. This kind of method seems true of most explorations 
of correspondence between image and sound through interaction. 

According to this approach a perfect system might be one that 
is flexible enough to generate an almost endless series of tonal 
variations, yet be controllable enough so that each one makes musical 
sense. In making such a system we primarily concern ourselves with 
two types of question, firstly about how data is mapped to sound, 
and secondly how we might operate the system, how we know 
what it will do under certain conditions and how we become skilled 
in its use. The ultimate assumption is that the system is a tool or 
instrument of expression, and that this knowledge will enable the 
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artist to control the system, so the he or she can articulate and 
communicate something about their feelings or beliefs. There are 
works that employ the opposite approach, ones that make the 
relationship between input and output problematic by creating 
systems that range from seemingly random, to ones whereby the 
users grasp some sense of what they are doing, but are unable to 
understand fully. It was known in advance that Yasunao's work 
embodied the polar opposite of these issues. Accepted knowledge 
gained in addressing such issues would be called into question (or 
even, totally ignored). Such work not only challenge ideas about art, 
technology and creativity but, in doing so, presents new perspectives 
on sound and image correspondence, generative systems and human 
computer interaction in the widest sense. 


A New Instrument for Interaction 


The interaction device made during the residency uses the Soft- 
board upon which the artist draws a series of strokes. The Soft-board 
sends information about pen color, and pen position to software 
that is used to synthesize sound. Projected onto the soft board is a 
sequence of video images selected by the artist. As the artist draws, 
the video image advances frame by frame. Data is taken from the 
x-y co-ordinates, the speed of movements, and mapped to synthesis 
parameters. The fundamental frequency of each sound generated is 
determined by the brightness of the pixel. Each pixel has brightness 
that generates frequencies in the range Ohz to 10khz. The brightness 
value is also used to multiply the amplitude of the sound, which 
results in distortion. When the pen is placed on the surface, the 
volume is switched on, and when the pen Is lifted off, the volume is 
silenced. Thus, sound is only heard when the pen is on the surface. 
When the pen is placed at the outer extremes of the board along the 
x axis (to the far left and far right) only the fundamental frequency is 
present. As the pen moves closer to the center the harmonic content 
is changed, the first harmonic is introduced, then the second and so 
forth. The speed of movement in the x axis (horizontal) is used to 
determine the volume of sound. Slow movements in this direction 
create quiet sounds, and fast movements create harsher sound. At 
the very center of the soft board drawing area, a small square is 
defined. When this area is drawn in, it produces white noise. 


The Artist's Viewpoint on the Collaboration 


".| was lucky enough to have been assisted by Mark Fell who had 
known my previous work and understanding my ideas suggested to 
me to use Soft-board for the visual to sound conversion. The system 
of SoftBoard has detected movements of hand drawing with the 
markers on the board with the accuracy of pixel by pixel, then the 
drawn lines are converted to binary data and stored in a computer 
for processing. 


The program that uses Max/MSP enables it to make parametric 
variants through scanning lines of video image from the harmonics 
we create beforehand. Also, it makes possible tracing positions of 
moving hand with a marker to coincide the sound distribution of 
pitches and timbers. So a stroke of calligraphy or tracing outline 
of an image invokes great variety of sounds. Soft-board has 
advantages over other devices | have used, which are: 


First, it makes possible to do in live performance situations, by 
combining with Max/MSP. Second, when | have created pieces | 
mentioned | had to transform the text into images, which is only 
possible with pictograms like Chinese characters. With Soft-board | 
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don't need to transform Chinese characters into images but simply 
draw calligraphy on the board. Also, it enables any text into sound. 
In only five days it was quite an achievement”. 

Conclusions 


In the Yasunao Tone residency, a highly challenging view of human- 
computer interaction was taken. It led to the development of a quite 
novel interaction device that formed the core base of a new creative 
performance artwork by Tone. The nature of the study undertaken, in 
which the roles of artist, technologist and researcher were allocated 
equal place, was highly effective in stimulating innovation. As well 
as enabling valuable research into the creative use of technology, 
the approach can be used to stimulate the creative development of 
technology. 
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A 3-D Metaphor tor Software Production Visualization 


by Thomas Panas, Rebecca Berrigan, and John Grundy 
ABSTRACT 


Software development is difficult because software is complex, 
the software production process is complex and understanding 
of software systems is a challenge. In this paper we propose a 3D 
visual approach to depict software production cost related program 
information to support software maintenance. The information 
helps us to reduce software maintenance costs, to plan the use of 
personnel wisely, to appoint experts efficiently and to detect system 
problems early. 


Section 1: Introduction 


Software engineers who are to maintain, extend or reuse a 
software system must understand it in the first place. Obtaining 


others. It is also likely that the cost of related aspects in a system 
such as defect levels or the effort required to design, develop, test 
and maintain a software product gather more attention in a business 
context focused on software production. 


Therefore, to simplify software complexity and increase program 
understandability while correlating with cost profiles across the 
product, reverse engineered programs must be effectively analyzed 
and visualized. Since developers think and perceive information 
differently, a tool must support user specific metaphors and 
views. Currently, metaphors and views within reverse engineering 
are restricted to depict program specific data only, i.e. they show 
merely information retrieved from code or data analysis. However, 
it is important and necessary to establish cost related analyses and 
views, which give answers to questions like: Which components are 
never executed? Which components are often changed? On which 
components do my developers currently work? Is the architectural 


figure 1: 3-D City 


this understanding of an industrial size system is often a time 
consuming process since most legacy systems are usually sparsely 
(or inadequately) documented. Estimates of the proportion of 
resources and time devoted to maintenance range from 50% to 
75% [1, 2]. The greatest part of the software maintenance process, 
in turn, is devoted to understanding the system being maintained. 
Fjeldstad and Hamlen report that 47% and 62% of time spent on 
actual enhancement and correction tasks, respectively, are devoted 
to comprehension activities [3]. 


Due to the above reasons, reverse engineering of legacy systems 
has become a vital matter in today's software industry. However, 
increasing program understandability is not a simple task due to 
the complexity of software systems. It is assumed that software 
complexity is related to the commonly accepted empirically derived 
features of software like code size, coupling, depth of inheritance, 
McCabe complexity related to logical paths and data flow, amongst 


structure of my system obsolete and hence needs to be refactored? 
Therefore, within this paper we present an idea that helps system 
maintainers and managers to e.g. decide, which components in a 
system are superfluous, which ones cause high cost due to frequent 
changes and whether the entire system needs to be revised. 


In Section 2 we give a brief introduction to program visualization 
for software maintenance. In Section 3 we present an example, 
showing a 3D city metaphor for program visualization. On top of 
this picture, we present a code production focussed metaphor in 
Section 4. Related work is summarised in Section 5. A discussion and 
future work is presented in Section 6. Finally, the paper is concluded 
in Section 7. 


Section 2: Program Visualization 


Visualization is the presentation of pictures, where each picture 
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presents an amount of easy distinguishable artifacts that are 
connected through some well defined relations. Visualization itself 
has a number of specialized foci [4]. However, in this paper we are 
merely interested in program visualization, dealing with static and 
dynamic code visualizations. Within program visualization, the scale 
and complexity of software are pressing issues, as is the associated 
information overload problem that this brings. In an attempt to 
address this problem the following concepts are considered to be 
important [5]: 


e Abstractions Program representations resulting from program 
analysis contain already for middle size programs enormous 
quantities of information. Consequently, to be able to adequately 
represent and comprehend the most relevant program information, 
the information assembled needs to be focused. There are basically 
three techniques to focus information [6]: Information abstraction, 
compression and fusion. 


¢ Metaphors The mapping from a program model (lower level of 
abstraction) to an image (higher level of abstraction) is defined 


Dynamic Scale. However, most vital for successful program 
visualization is the retrieval of necessary data for visualization and 
the availability of a suitable metaphor. 


3 A 3D City Example for Program Development Visualization 


Metaphors, when depicting real worlds and establishing social 
interaction [16], especially in virtual reality [17, 18], become very 
important. Essential is therefore the choice of metaphor to improve 
the usability of a system. One fundamental problem with many 
graphic designs is that they have no intuitive interpretation, and 
the user must be trained in order to understand them. Metaphors 
found in nature or in the real world avoid this by providing a graphic 
design that the user already understands. When illustrating a reverse 
engineered architecture, it is important for the understanding of a 
program that the final picture is adjusted for the individual [19, 20]. 
Therefore, we are currently developing a visualization architecture 
that allows all kind of metaphors to increase individual program 
understandability. 


figure 2: 3-D City from above 


through a metaphor, specifying the type of visualization. Most 
visualization techniques and tools are based on the graph metaphor 
(including the extensive research on graph layout algorithms). Other 
initiatives are the representation of programs as 3D city notations 
[7], solar systems [8], video games [7, 9], nested boxes [10, 11], 3D 
Space [12], etc. 


e Visualizations It is not feasible to depict all kinds of program 
model phenomena in just one picture when the model carries 
too much information. Therefore each program model is depicted 
through various views, guaranteeing that the right subsets of 
objects and their relations are depicted and understood. Most effort 
to solve software complexity was put into different visualization 
forms, mainly the graph metaphor, including UML diagrams, to 
depict various program class and architecture views [4, 13, 14, 15]. 


The success and quality of any visualization depends on many vital 
features [4]: Animation, Metaphors, Interconnection, Interaction, 
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Within our unified recovery architecture [6, 21], the selection of 
metaphor can be undertaken depending on the user's requirements 
and focus in visualizing program information. Currently, we are 
implementing a 3D City metaphor to our architecture to support 
program understanding within three dimensions - this could be 
changed for an alternate world in a user dependent fashion. Figure 1 
shows a screenshot of the running example, where buildings denote 
components (mainly Java classes) and the city itself represents a 
package. Different metaphors between the source code and the 
visualization are possible, i.e, components must not always be 
mapped to buildings and packages to cities. Other compilations are 
thinkable, where, e.g., buildings are mapped to methods. To support 
the user with an intuitive interpretation of a software city and to 
increase the overall realism of the metaphor, we added trees, streets 
and street lamps to the figure. 


Further, the figure illustrates both static as well as dynamic 
information about a program. From a static point of view, the size 


of the buildings give the system maintainers an idea about the 
amount of lines of code of the different components. The density of 
buildings in any certain area shows the amount of coupling between 
components, where the information for this can easily be retrieved 
from metric analysis. The quality of the systems implementation 
within the various components is visualized through the buildings 
structures, i.e. old and collapsed buildings indicate source code that 
needs to be refactored. 


From the dynamic standpoint, cars moving through the city indicate 
a program run. Cars originating from different components leave 
traces in different colors, so that their origin and destination can 
easily be determined. Dense traffic indicates heavy communication 
between various components. Performance and priority are depicted 
through the speed and type of vehicles. Occasionally exceptions 
occur, where cars collide with other cars or buildings, leading to 
explosions. 


A satellite view of our analyzed system can be seen in Figure 2, where 
cities (packages) within the architecture are connected via streets 


4 A Metaphor Designed to Highlight Production Information 


Program visualization can provide a large reduction in effort 
associated with program understanding. However, in order to 
increase the reactivity of a development environment with respect 
to business realities, highly specialized, visual information should be 
delivered in a timely fashion to those people who can effect greatest 
impact. 


Developers are mainly interested in information about functional 
and non-functional issues of a program, e.g., which components 
do | have to modify in order to change the security aspect of my 
system? Project managers need to know to which parts of the 
program their team is allocated to, answering questions like, can 
we meet the next deadline? Additionally, managers and vendors are 
interested in hot spots, which means components that have been 
modified frequently, indicating the high cost areas of the systems 
maintenance. Designers and maintainers are more interested in the 
overall structure of the system, indicating places which are of heavy 
or little use. This helps to decide which components to remove or how 


(two-directional calls) or water (unidirectional calls). Information 
between cities (packages) is passed via boats and vehicles. Again, 
dynamic as well as static information is illustrated. Clouds in that 
figure cover cities that are not of current interest to the user and 
hence are hidden. 


The general idea is to fly interactively through a reverse engineered 
software system and depict it within a 3D city. The user must have 
full freedom in zooming and navigating through the system, and 
even be able to perceive the system not only on the usual computer 
monitor, but also within a virtual environment. However, the figures 
represent, so far, only static and dynamic information about a 
reverse engineered software system, which is received from the 
programs source code. For maintenance, however, we wish for more 
information to answer cost related questions, like where does the 
main maintenance cost occur in the system. In order to answer 
these questions in a picture, we present next a software production 
cost related 3D city metaphor. 


to restructure the system in order to achieve a higher performance 
or reliability. Although Figure 2 and Figure 3 help to visualize various 
static as well as dynamic aspects of a reverse engineered system 
and hence increase the understandability of the system, they do not 
provide support for the various additional demands that developers, 
designers, vendors and project managers have. Therefore, we apply 
a cost focused metaphor over the 3D city metaphor, to visualize 
additional business related information of a system. Figure 4 shows 
our approach. 


In the figure we illustrate various additional aspects relevant for 
different types of stakeholders. 


e Work Distribution. The components currently being modified by 
the staff are indicated in yellow with the respective names. This 
gives an idea about the progress of the maintenance or development 
team and can help to estimate the total time needed. 
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e Recycling. In brown, those parts of the system are illustrated, 
which are no longer needed. These parts have not been used for 
quite a while and increase only the total complexity of the system. 
Those parts should be further investigated for removal. 


© Hot Execution Spots. Components with frequent execution are 
indicated by a surrounding fire. In order to increase the reliability 
and performance of a system it is beneficial to investigate these 
components carefully. 


¢ Aspects. Aspects, from Aspect-Oriented Programming [22, 23], 
can be depicted in various colors. In the figure here, distribution is 
shown in blue and synchronization in green. These spots help out to 
point out functional or non-functional cross-cuttings that can be 
investigated and further changed by experts. 


¢ High Costs. Buildings with flashes indicate frequent component 
modifications. The result is an increased maintenance cost and 
hence also an increased cost for the entire software project. 


the concept lattice. For reverse engineering, component detection 
can be optimized by interactively illustrating the changing picture of 
the city while the concept lattice of the analysis is being changed. 


Finally, the navigation within the city is important in order to 
present the 3D city to the user in the most natural way. Software 
maintenance should not necessarily require reading millions of lines 
of code. Navigating through the city from top or even from inside a 
car (program counter), might make maintenance a game. Complex, 
huge software systems could be comprehended much faster. 


5 Related Work 


At present there exists some work on visualizing source code in three 
dimensions, however little effort was made to depict production 
relevant information in 3D. Knight and Munro [7] describe what 
they call the ‘software world’ - a system being visualized in the 
world. Cities represent source files that that may contain one or more 
classes which themselves are represented as districts and finally 


The list above is not complete. Many more production related issues 
exist that could be depicted. Vendors and managers might see the 
quality of a software system at once, by having a short look at the 
3D city. Lots of fire, flashes and mud indicate high cost areas of 
code and unacceptably high risk regarding the ongoing health of the 
system. Developers might benefit from business process, use-case, 
control flow, data-flow, and event-based visualizations. For example, 
a dynamic picture of events occurring on various positions in the 
city and being handled on other positions (indicated by e.g. colors) 
can aid the understanding of event-based software development. 
Another example might be an animation of cars driving between the 
different cities, marking them in color, to indicate which components 
are used when and for which use case. 


Further, static and dynamic source code analysis can be illustrated, 
showing a changing city, depending on the type of analysis applied. 
For example, when applying the concept analysis, which computes 
maximum sets of objects using the same set of attributes, different 
components can be formed, depending on the user interaction on 
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methods are shown as buildings. Here colors are used on buildings 
to encode the type of methods: public or private. However, the focus 
of this paper is the application of virtual reality technology to the 
problem of visualizing data artifacts and not production related 
visualization. 


In [24] the authors present a case study, which shows the possibility 
of deploying 3D visualizations in a business context. In the case 
study a reusable collection of behaviors and visualization primitives 
is presented (3D gadgets), with the intention to deploy visualization 
components in a software architecture. However, the paper 
focuses on business process visualization to validate requirements 
and to acquire feedback on design-tradeoffs and not on program 
understanding or software maintenance. 


6 Discussion and FutureWork 


As mentioned above, the 3D city is an example metaphor for our 
unified visualization architecture that is currently being developed. 


The architecture supports any kind of reverse engineering, 
following three basic steps: analysis of a program, focusing (which 
means filtering, folding, fusing, etc. of program information), and 
the visualization of retrieved source code information. For the 
visualization, a mapping between the data representations and the 
final picture is needed. Hence, we need metaphors. 


Currently, we are about to finish our unified recovery architecture 
that allows us to plug in various visualizations, including metaphors 
and layout algorithms, providing the best fit system view for an 
individual user. The step to follow is to implement the 3D city 
metaphor in, e.g., OpenGL4Java, in order to perform tests and 
evaluations. The present study about the 3D city is based on our 
3DStudio Max implementation. 


The unified recovery architecture permits the visual correlation of 
code structure information and development house operational 
information in a single viewscreen. The choice of metaphor can also 
be altered depending on the relative importance of each aspect of the 
code (or cost related to the code) to the viewer at hand. Once these 
tools are robust and readily configurable the test will be to undertake 
a medium scale industrial trial where scalability, performance and 
time constraints will test the ability of the visualization tool to 
provide valuable information to the organization. 


7 Conclusion 


In this paper, we have illustrated our 3D city metaphor that we are 
currently developing for our unified recovery architecture. Further, we 
have discovered the need to depict more than just static or dynamic 
program information for increasing a programs understandability 
and maintainability. We need to depict production cost related 
information of a system that shows the right stakeholder the right 
information using minimal time and effort. In general we have tried 
to present a vision where software maintenance must not always be 
labor intensive, trying to understand and alter undocumented legacy 
code, but also fun, when navigating through a software city in 3D. 
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Pointing the tinger; 


a role for hybrid representations in VR and video? 


by John Counsell 
Introduction 


Recent experience gained during an EU funded project called 
Valhalla ' showed that linking web based 3D modelling with real- 
time web video is an effective technique for explaining the content 
of the video images.’ In the project a hyperlinked 3D model was 
located alongside the real-time video image, on a web page, for 
public information in a virtual historic gardens tour. Software was 
written to: interactively point, pan and zoom the camera across the 
web (and script for repeat performances); and slave the 3D model to 
the video camera controls, in order to continually adjust the field of 
view and focus in the 3D model to match the video image. 


This is what we have termed ‘hybrid representation’ using real-time 
video synchronised with a matching moving viewpoint in the 3D 
model. The video serves to provide the sense of animation, life and 
reality that is difficult to achieve in VR. The modelling provides a 
measurable 3D spatial framework and a means of automating 
hotspot access to more abstract information. Interaction by the 
user on the web, clicking on a hyperlinked object in the model, 
instigates a database query that returns further information about 
or associated with the object. In the Valhalla project this was related 
to plant information, architectural history and the like. In an Urban 
Modelling context, it might for example be related to information 
about shops, businesses, goods for sale, occupancy and contact 
details, or tourism and history. 
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The user interactively searched the 3D model to find information 
about matching objects in the video, clicking on each object that 
engaged interest, to view the database information associated with 
it. However if the user was not exploring the image, but searching 
for the location of a specific known object within the field of view, 
the process was more ‘hit and miss. We did want to make the 
process even more intuitive by invisibly superimposing the model as 
overlaid dynamic “hotspots” on the video image, but this would have 
required a longer project and more resources than were available, 
and so still forms a future stage of development. 


We developed a server application to search the spatial information 
for instances of desired objects based on search terms typed into the 
web interface.? Instigating a search in the database would reference 
multiple different locations of the same plant or other object within 
the visible 3D scene, and could return parameters that focused the 
field of view on each instance in turn. This we have referred to as 
virtually ‘pointing a finger’ at the object of interest. 


This paper focuses on the real-time interactive use of a realistic 
urban model as an urban information system, based on an underlying 
database of spatially referenced information. Another way of 
expressing the issue is to ask how can a realistic urban model itself 
act as a navigational guide, focussing attention in a natural way 
and or drawing the attention forward from one point of interest to 
another, without the insertion of avatars, symbols, or labels in the 
scene? 
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figure 1. Clicking on hyperlinked objects in the synched 3D model on the left return text below about the plants from the database 
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Pier et 
Figure 2. Bristol Harbourside 3D Web model 


3D Urban Information Systems 


The first virtual city model with high levels of realism and interactive 
navigation was probably the 1978 Aspen Moviemap.* Its development 
was influenced by the observation that people find it easier to 
remember things related to actual locations. This observation 
originated from the use of mind-maps® for memory training in 
classical and Renaissance literature, which used metaphysical 
devices such as the memory theatre®, walked through to retrieve 
the associated data. In 1999 we helped develop a prototype web 3D 
model for a large urban area of a capital city, where clicking on the 
links on the buildings opened websites for the individual businesses 
that occupied the buildings. It was a 3D approach to digital 
‘yellow pages. Navigation in the prototype was by street, address, 
and ‘reading’ the signboards or looking at the shop-fronts, before 
finding out more about the internal organisation by examining the 
WebPages. There remains the issue of drawing attention (while 
absorbed by interaction with or immersed in virtual urban space) to 
unknown information, or to desired information that is related to a 
place or location that is not known but could be shown. 


Avatars serve a useful purpose in acting as surrogates or symbols for 
individuals in real-time encounters in shared community spaces. It 
may be considered that this is a current compensation for the lack 
of bandwidth to enable actual video or real-time 3D scans of these 
individuals’. Avatars have also been extensively used as scripted or 
automated ‘natural’ guides in interpreting VR recreations of past 
cities and environments. These might be a distraction from the use 
of the realistic urban model as an information space, and are limited 
by only directing attention to one location in the geometric model 
at a time in a sequential rather than a pattern recognition process. 


In the Smartkom project,’ information has to date been ‘published’ 
from the spatial data in the database as voice prompts or text labels. 
The voice prompts are in effect a sequential approach to drawing 
attention and we would suggest are not a full substitute for visual 


The text labels 


1 are overlaid on the 
The Cathedral images derived from 
the 3D model obscuring 
significant portions, 
which we suggest is 
undesirable. 
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Drawing attention 
to parts of the video 
image 


We considered how 
best to simultaneously 
‘highlight’ the location 
of a number of objects 
of interest in the video 
image. It would not be 
difficult to tag multiple 
instances simultaneously 
in a wide-angle view 
for more instant, less 
sequential, pattern 
recognition. We believe 
that it remains difficult 
to determine how most 
effectively to ‘highlight’ 
these instances for immediate recognition in a wide field of view 
without further research. 


| Se ete 


It has been argued that ‘highlighting techniques, like blinking, 
brightening, reverse video, can be used to focus the attention to 
specific items among thousands of items: While multiple objects of 
interest in the 3D model could be simultaneously false coloured or 
otherwise distorted to attract attention, this would tend to obscure 
those very portions of the image to which attention is desirably 
drawn, particularly if the 3D were to be overlaid transparently on the 
video image, as expressed above as a longer term goal. Nevertheless 
portions of that model could be made more visible to act as 
overlaid highlights. We considered that an outcome that obscures is 
undesirable, since a particular value of such models is held to be the 
convincing sense of presence that they convey. Although there are 
other instances where it is useful to distort a photo-realistic image 
to distinguish specific portions, for example it would be useful 
to distinguish between certainty and uncertainty, reality and the 
theoretical, in archaeological presentations or planning proposals. 


Senses other than the visual could be used in VR space, for example 
hot for closer, cold for more remote, which might effectively be linked 
to eye-tracking, however this still seems a hit and miss approach to 
drawing attention. 


There is an implicit assumption in much VR that movement is linear 
and corresponds with movement in the real world. In the film world 
non linear conventions have developed that tell effective stories and 
draw attention using a filmic visual language that is now accepted 
by the film and TV viewing public world wide. An example is that 
of montage - the juxtaposition of shots as opposed to sequential 
procession - through editing, authoring or interaction.'° Such a 
narrative juxtaposition may form an acceptable alternative in those 
situations where a single wide angle view cannot serve, however 
automating a technique of generating a filmic and therefore 
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‘acceptable’ series of montage shots of a 3D digital urban model 
would be in itself a demanding task. 


Perceptual Artists 


Perspective projections or orthographic projections, such as 
isometric, axonometric, oblique, have all been used in 3D games. 
(The isometric projection was used extensively in Roman, Byzantine, 
Persian paintings and 18th C Japanese woodcuts). The disadvantages 
of a wide-angle perspective view are that objects in the foreground 
are more dominant since they occupy a larger portion of the scene. 
Conversely the advantages of an isometric projection are that objects 
in the back or middle ground are given equal ‘weight’ to those of the 
same size in the foreground, so a single wide angle viewpoint could 
draw unbiased attention to a widely distributed range of objects of 


interest. 


Figure 3. Icons serve to explain which is an office and which is a house, but obscure. Image created in explora- 
tion of Bristol Model in 1996. 


In addition to the suggested effective use of isometric projections, 
perceptual artists have used techniques in the last century that 
are now being shown to be both readily understandable, and more 
closely related to cognitive perception, than either perspective or 
orthographic projections. Future development of computer based 
tools to implement such techniques are now being forecast. These 
may prove the most useful approach of all to drawing attention to 
objects of interest without losing meaning or obscuring parts of the 
image. Van Gogh, for example, painted scenes in a manner which 
was not based on true perspective, yet both very meaningful and 
with a clear focus (such as his famous painting of the church of 
Notre Dame at Auvers-sur-Oise)." The painting is not fully based on 
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euclidean perspective, but draws the attention in a striking way. It 
may be argued that there is much to learn from perceptual artists in 
how to draw and capture the attention. 


At a recent conference in Glasgow on visual perception (ECVP 2002) 
an exhibition organised by an artist, John Jupe, challenged orthodox 
views on perception.'? He presented a number of paintings based 
on an analysis of perception in the works of great artists such as 
Cezanne. In his paintings and analysis he demonstrated images 
which contained a fixation area, within which the representation 
was painted in a manner which was three dimensional with greater 
detail and enhanced colour; and the remainder of the painting was 
in a manner which is flatter and less 3D and addresses peripheral 
vision." His claim was that such images address the viewer in 
the manner the viewer perceives reality, and so are closer to the 
normal manner of visual perception. Such an approach, if it could 
be effected in an automated manner, might be very appropriate 
in drawing attention to the 
object or area of interest (in his 
terms the fixation point) while 
rendering the remainder of the 
image as a flatter, less detailed, 


but nevertheless meaningful 
context. 
Conclusion 
This paper describes = an 


investigation into an area of 
visual information display that 
has not been greatly explored. 
There is some evidence that 
large area models are still not 
commonplace in VR. Much of 
the focus in VR could be argued 
to be on the techniques and 
the realism rather than on the 
database of information that 
could underlie such a model and 
on expression of the results of 
database searches in the model 
within the VR scene. Generally 
speaking, searching the VR model 
requires interactive exploration 
by the user. It remains more 
difficult to quickly identify the 
location of an object of interest 
in the VR scene, if a search is 
run in the database to return the 
location of objects in the scene. 
This paper suggests that there are a number of instances where it 
would be useful to distort a photo realistic image to draw attention 
to specific portions, ranging from the representation of uncertainty 
in archaeological presentations or planning proposals, to the 
identification of the location of an object of interest in a video 
clip or live video frame. Indeed, multiple techniques could well be 
combined for greater effectiveness. If the 3D ‘hotspot’ model were 
overlaid transparently on the Video image, as is the goal of the next 
stage of the Valhalla project, then these techniques will be required 
in order to effectively ‘point the finger’ at objects of interest in the 
underlying video. The approaches indicated in this paper require 
more exploration to determine ease of use and speedy absorption of 


Figures 4 (above) and 5 (below).. Images created in exploration of Bristol Model in 1996. Becons 
above (searchlights_ obscure all but the objects of interest, while edge highlights (below) separate 


the object from its immediate setting 


data without changing the scene beyond recognition. 
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Shape of the Story: Story Visualization Techniques 


by X. Zeng, O. H. Mehdi and N. E. Gough 
1. Introduction 


The basic plot of a story does not change very much but the ways 
of storytelling are challenged and changed by text, image and sound 
technology developments. Unlike text-based format storytelling, the 
readers and listeners convey the words into a virtual environment 
in their own mind. Story visualization focuses on using visual 
metaphors to represent non-visual contents and relationships of the 
story [11]. Picture books, theatre, television and movies, etc have 
one thing in common: they extract abstract information of the story 
into concrete visual scenes to enhance the entire experience and 
evoke emotions of the audience. The multimedia computer with its 
capacity for handling text, sound and images has a profound effect 
on this process [5]. The increasing development of Three Dimensional 
Virtual Environment (3DVE) as a popular concept has produced a new 
medium for storytelling. The 
power of realistic images and 
interaction give participants | Santance 
more freedom to_ interact 
with the story, but how to i : 
build this kind of VE still lies 
in the domain of highly skilled 
professionals. Even the most 
expert 3D animators need to 
spend considerable time to 
create interactive 3DVE that 


images or create scenes. In contrast, we describe a methodology that 
enables non-professionals to generate a 3DVE by using simplified 
story-based NL input. Furthermore, the system allows the user to 
manipulate the visual features of VE in real-time. 


3. System Overview 


Our system is based on the system developed by [11, 6] which 
consists of several modules that are written in Java, XML (eXtensibe 
Markup Language) and VRML. The system adopted in this work is 
shown in Figure 1. The sentence is tagged and output to a knowledge 
base system and then the output XML-tagged parse tree is interpreted 
into a semantic representation. Finally the semantic representation 
is converted into a set of parameterized data by a Descriptionary (a 
dictionary that uses XML based word frames to parameterize visual 
or describable words), which can be used by the graphic engine to 
generate the atmosphere or objects and ultimately to construct a 

virtual environment. 


3.1. Language Engine 


We argue that it is 
unnecessary to involve a 
complicated description for 
creating a 3DVE by story- 
based NL input, we should 
just focus on the direct use 
of visual features of the 


could be described in just a 
few sentences from a story. The 
awareness of possibilities for 
integrating a Natural Language 
(NL) interface and computer 


Graphic 
Engine 


Knowledge 
Base 


"(Language 


environment, such as colour, 
size, material, direction and 
spatial features, etc. Hence 
we may start with children's 
picture books to understand 


graphics by incorporating 
knowledge of human-to- 
human interaction has been 
considered in improved VE 
tool user-interfaces. NL is the 
common tool that people are most familiar with as it allows them to 
describe visual ideas and visual scenes in a straightforward manner. 
Generation of a 3DVE by using NL input does not require substantial 
training and could speed up the whole process. In this paper we 
introduce an approach that enables non-professionals to create 
an interactive 3DVE through manipulating Visual Features of the 
aspects of environment by using story-based NL input. 


Engine 


Figure 1. The overview of the system 


2. Related Work 


Research on automatic construction of visual scenes based 
on NL input can be dated back to the early 70s since the SHRDLU 
system [9]. SPRINT [10] and Put [1] are language-based systems that 
focus on spatial relationships to reconstruct 3D models of the world, 
and output the corresponding image on the graphic display. CarSim 
[3] implemented NL descriptions to generate animation scene and 
replay of accidents. More recently, [2] presented a system called 
WordsEye for automatically converting text into representative 3D 
scenes by using linguistic analysis and depiction techniques. These 
efforts either focus on implementation of NL to represent the still 2D 
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the structure of the story. 
Moreover, it includes 
constraints and context to 
help avoid ambiguity. This 
affords the ability to deal with 
limited syntax and semantics by ignoring many of the problems of 
unknown or meaningless vocabulary and unconstrained grammar. 


NLP uses known data about words and predefined rules that 
show authorized word structures to give the meaning of what was 
said to enable communication between people and computers [8]. 
Because our main goal is to represent 3D graphics by text input 
rather than create some new language algorithms, currently 
available computer linguistic techniques will be sufficient to solve 
the problem of semantic presentation. XML has been used as the 
primary data structure in the language engine. XML and the Java 
based language engine have two tasks in this system: one is to find 
matched VRML objects from text and the other is to translate object 
depictions (e.g. adjective, prepositions and verbs) into parameterized 
data definitions for use by graphic engine to create a 3DVE (i.e. 
object's attribute, spatial relationships and actions or events). 


Our language engine consists of language tagging and knowledge 
base modules. Each processes the text in a different way and the 


Figure 2. Construction of language engine 


output is used by the next components without loss of initial 
information. The construction of the language engine is shown in 
Figure 2. The first component is the language tagging, which utilizes 
annotation rules to perform lexical lookup, and finds semantic 
role-definitions of words and phrases in the sentences. The second 
component is the knowledge base, which uses grammatical rules 
to map tagged output into semantic representations and finally 
parameterize semantically elements into data definitions. Some 
basic knowledge of grammar and understanding of the structure 
of sentences can be added for semantic representation and can 
be translated into the following elements, Time and Atmosphere, 
Location, Object to correspond nouns; Number to article or digit; 
Attribute to adjectives; Action to Verbs and Position to prepositions. 
These represent the various meanings and relations inherent from 
the input sentence. 


The next step involves converting the various semantic elements 
into a parameterized data definition, and defining inferences of the 
visual features through object matter and context from the input 
text. An extendable dictionary called "Descriptionary" was created 
and predefined in a knowledge base module. The Descriptionary uses 
XML based word frames to parameterize a few "visual or describable 
words" such as nouns, adjectives, adverbs and prepositions into low- 
level data to communicate with the graphic engine for manipulating 
the atmosphere or objects in the VE. 


3. 2. Graphic Engine 


Environment presents the information of where and when the 
story happens and it is the setting and stage of the story. VRML is 
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Figure 3. VRML scene graph structure 


used as our primary object format and rendering engine since it 
is a scene graph. Its text based file format and the construction 
show that it could be the perfect platform to help us address 
the visual features of the VE by NL input. Figure 3 presents the 
VRML based scene graph structure and the four types of VRML 
nodes are classified by their function. These nodes correspond 
to relative describable nouns, adjectives, preposition, etc - the 
visual features of the VE. 


While VRML is not a general-purpose programming language 
and Java is not a 3D presentation language, integrating the two 
languages gives interactive 3D graphics, complete programming 
capabilities and extensive support for building large-scale virtual 
environments. The External Authoring Interface (EAI) [4] defines 
a Java interface for direct communication with the VRML world. 
This interface contains functions such as searching existing 
tagged and named nodes, sending an event into a node of VRML, 
etc. The graphic engine consists of two components: a Java 
applet that forms the input interface and the VRML world are 
linked through the EAI (see Figure 4). The Java applet provides 
interaction with the language engine. A number of VRML format 
objects have been classified and contained in the object database 
that are all built to their default size and in neutral states. For 
our graphic engine, the system has to specify and translate 
several parameters of elements that are filled in by default value 


Figure 4. Components of graphic engine 


in the language engine (i.e. atmosphere of the environment, the 
coordinates of objects, etc.) as a Java string to call corresponding 
elements or manipulate them by sending events into an existing 
VRML scene to construct a VE accordingly. 


4. The Example 


A simple example that demonstrates how to construct 3DVE 
using the method described above is shown in Figure 5. 


It is a sunny summer mid day. In a green field, there is a blue 
table on the ground. There is ball on the table. The ball is 
made of glass. 


These sentences are interpreted and presented by following 
semantic presentations: 


A. Time of when the story happens, It is a sunny summer 
mid day. 
Time: summer (mid day) 
Atmosphere: sunny 
This determines the atmosphere of the environment, such as 
light condition, sky of the color, etc. 
B. Place where the story happens, In a green field. 
Location: field 
Attribute: 
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Colour: green 
C. Objects on the field: 
Object1: table 
Attribute: 
Colour: blue 


Consider the last sentence to understand how the Java applet 
interacts with the VRML world through the following steps: 


1. Get a reference to the VRML browser 
Browser browser = Browser.getBrowser(this) 
2. Find the node named “BALL” 
Node ball = browser.getNode("BALL"); 
3. Get the “set_transparency" Eventin field 
EventInValue_changed transparency = 
changed) ball.getEventin("set_transparency"); 
4. Add the value to the transparency field 
transparency.setValue(0.3); 
5. Refresh Browser. 


(EventInValue_ 


If the entered sentences do not contain 
detailed information, such as viewpoint, 
actual table and ball size, colour of the 
ball etc the system uses the default size 
as a solution. 


5. Conclusion and Future Work 


An approach that allows non- 
professionals to generate a 3DVE by 
using story visualization technology has 
been introduced. Story is the primary 
potential source for applying visualization 
techniques and the scene graph based 
VRML provides the opportunity to 
manipulate and modify properties of the 
environment in real time. We believe that 
integrating NL and 3D graphic techniques 
provides a flexible and easy way for non- 
professionals to generate and interact 
with the VE in real time as compared to 
traditional 3D packages. This project is 
still under development. We find there 
are still various aspects of the system 
that need to be improved. Future work 
includes: the option of voice input could 
be more useful instead of text typing; 
research will also take place to develop 
a event engine and how to embed a BBAI 
(behavioural-based Al) character to carry 
out the describable tasks in the domains 
of time, space and motion in VE. 
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